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What  is  NIST? 


•  U.S.  National  Institute  of  Standards  and  Technology 

•  A  non-regulatory  agency  in  Dept,  of  Commerce 

•  3,000  employees  +  adjuncts 


•  Gaithersburg,  Maryland  and  Boulder,  Colorado 

•  Primarily  research,  not  funding 


•  Over  100  years  in  standards  and  measurements: 
from  dental  ceramics  to  microspheres,  from  quantum 
computers  to  fire  codes,  from  body  armor  to  DNA 
forensics,  from  biometrics  to  text  retrieval. 


Four  atom 
quantum 


The  NIST  SAMATE  Project 


Software  Assurance  Metrics  And  Tool  Evaluatiorysg^ 
(SAMATE)  project  is  sponsored  in  part  by  DHS 

Current  areas  of  concentration 


-  Web  application  scanners 

-  Source  code  security  analyzers 

-  Static  Analyzer  Tool  Exposition  (SATE) 

-  Software  Reference  Dataset 

-  Software  labels 

-  Malware  research  protocols 

•  Web  site  http://samate.nist.gov/ 
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Software  Reference  Dataset 


Extended  Search  |J  Source  Code  Search 

Number  (Test  case  ID): 

Description  contains : 

Contributor/Author : 

Bad  /  Good  : 

Language  : 

Type  of  Artifact : 


Any... 


Any...  7 


Any... 


Status  .  Candidate  IZI  Approved  IZl 
Weakness  : 


Any... 


Code  complexity : 


Any. 


Date.  Q  Before  O  After 

(Format:  M/d  AO  - 


use  the  calendar  (next  icon) 

\  Search  Test  Cases 


Weakness  Code  Complexity 

□-Any... 

[+1-CWE-485:  Insufficient  Encapsulation 
&CWE-308:  Error  Handling 

0-CWE-389:  Error  Conditions,  Return  Values,  Status  Codes 
CWE-254:  Security  Features 
CWE-227:  Failure  to  Fulfill  API  Contract  (API  Abuse) 

CWE-019:  Data  Handling 
CWE-361:Time  and  State 
CWE-398:  Indicator  of  Poor  Code  Quality 
I— CWE-470:  Use  of  Externally-Controlled  Input  to  Select  Classes 
&CWE-465:  Pointer  Issues 
&CWE-41 1 :  Resource  Locking  Problems 
—  CWE-401 :  Failure  to  Release  Memory  Before  Removing  Last  F 
-GWE-415:  Double  Free 
-CWE-416:  Use  After  Free 
□-CWE-41 7:  Channel  and  Path  Errors 
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Public  repository  for 
software  test  cases 

Almost  1800  cases  in  C, 
C++,  Java,  and  Python 

Search  and  compose 
custom  Test  Suites 

Contributions  from 
Fortify,  Defence  R&D 
Canada,  Klocwork,  MiT 
Lincoln  Laboratory, 
Praxis,  Secure  Software 
etc. 
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Software  Facts  Label 

•  Software  Facts  should: 

-  Voluntary 

-  Absolutely  simple  to  produce 

-  Have  a  standard  format  for  other  claims 

•  What  could  be  easily  supplied? 

-  Source  available?  Yes/No/Escrowed 

-  Default  installation  is  secure? 

-  Accessed:  network,  disk, ... 

-  What  configuration  files?  (registry, ...) 

-  Certificates  (eg,  "No  Severe  weaknesses  found 
by  CodeChecker  ver.  3.2") 

•  Cautions 

-  A  label  can  give  false  confidence. 

-  A  label  shut  out  better  software. 

-  Labeling  diverts  effort  from  real  improvements. 
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Software  Facts 


Name  InvadihgAlienOS 
Version  1996.7.04 
E.'^pected  number  of  users  15 


Modules  5  48  3  Mod  u  le  s  f no  m  lib  rarie  s  4  102 


Vulnerability 


Cross  Site  Sicripting  22. 

65% 

Heflected  1 2 

55% 

Stored  1 0 

55% 

SQL  Injectran  2 

10% 

Butter  overflow  5 

95% 

Total  Security  Mechanisins  284 

1 00% 

Authentication  15 

5% 

Access  control  3 

1% 

Input  validation  230 

81% 

Encryption  3 

1% 

AES  256  bits.  Triple  DES 

Re  port  security  flaws  to:  ciwnmcyi(®mothership.milkyway 

! 

Q 

Total  Code  3.1415-^10  function  points 

100% 

9 

C  1.1-10  function  points 

35% 

g 

Ratfor  2.041 5if  10  function  points 

65% 

Test  Material  2  71 8>^  10®  bytes 

100% 

Data2. 69^10®  bytes 

99% 

E:<ecutables  27.18x10^  bytes 

1% 

Documentation  12  058  pages 

100% 

Tutorial  3  971  pages 

33% 

Reference  6  233  pages 

52% 

Design  S  Specification  1  854  pages 

15% 

1 

Libraries;  Sun  Javs  I  .5  runtime,  Sun  J2EE  1.2.2, 
Jakarta  log4j  1 .5,  Jakarta  Commons  2.1 , 

Jakarta  Struts  2 .0 ,  H  aro  Id  XOM  1 . 1  rc4 ,  H  u  nte  r  J  DOM v  1 


Compiled  with  gcc  (GCC)  3.3.1 


Stripped  of  all  symbols  and  relocation  information. 


Researching  Risky  Software 


•  Many  people  research  malware,  but  there 
are  no  widely  accepted  protocols. 

•  Biological  research  has  defined 
ieveis  with  associated  practices, 
safety  equipment,  and  faciiities. 

•  Some  approaches  are 

-  Weakened  programs  (auxotrophs) 

-  Programs  that  ALERT 

-  Outgoing  firewalls 

-  Isolated  networks 
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4tti  EdHion 


CLt  •  Nin 


Biosofely  in  Microbiological 
and  Biomedical  Laboratories 


bfrAfTMEttl  OY  HCftLIf  AYiU  HUhUrJ=  SHVUFi 


•  Assurance  that  software  is  less 
vulnerable  to  coming  cyberassaults 


Static  and  dynamic  analysis 

Static  Analysis  Tool  Exposition  - 
outcomes  and  2010  progress 
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Assurance  from  three  sources 


A  =  f(p,  s,  e) 

where  A  is  functional  assurance,  p  is 
process  quality,  s  is  assessed  quality  of 
software,  and  e  is  execution  resilience. 
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p  is  process  quality 


•  High  assurance  software  must  be 
developed  with  care,  for  instance: 

-  Validated  requirements 

-  Good  system  architecture 

-  Security  designed-  and  built  in 

-  Trained  programmers 
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5  is  assessed  quality  of  software 


•  Two  general  kinds  of  software 
assessment: 

-  Static  analysis 

*  e.g.  code  reviews  and  scanner  tools 

*  examines  code 

-  Testing  (dynamic  analysis) 

*  e.g.  penetration  testing,  fuzzing,  and  red  teams 

*  runs  code 
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e  is  execution  resilience 


•  The  execution  platform  can  add  assurance 
that  the  system  will  function  as  intended. 

•  Some  techniques  are: 

-  Randomize  memory  allocation 

-  Execute  in  a  “sandbox”  or  virtual  machine 

-  Monitor  execution  and  react  to  intrusions 

-  Replicate  processes  and  vote  on  output 
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Software  analysis  is  vital 


•  Benefits  are: 

-  Provide  feedback  to  development  process 

-  Build  product  assurance  when  process  is  less 
visible 

*  contractors 

*  open  source 

*  legacy  software 

-  Confirm  minimum  quality  for  execution 
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•  •  • 


Analysis  is  like  a  seatbelt 
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•  Assurance  that  software  is  less 
vulnerable  to  coming  cyberassaults 

•  Static  and  dynamic  analysis 


•  Static  Analysis  Tool  Exposition  - 
203Houtcomes  and  2010  progress 
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Comparing  Static  Analysis  with 
Dynamic  Analysis 


Static  Analysis 

•  Code  review 

•  Binary,  byte,  or  source 
code  scanners 

•  Modei  checkers  &  property 
proofs 

•  Assurance  case 


Dynamic  Analysis 

•  Execute  code 

•  Simuiate  design 

•  Fuzzing,  coverage,  MC/DC, 
use  cases 

•  Penetration  testing 

•  Fieid  tests 

a 
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Strengths  of  Static  Analysis 


•  Applies  to  many  artifacts,  not  just  code 

•  Independent  of  platform 

•  In  theory,  examines  all  possible 
executions,  paths,  states,  etc. 

•  Can  focus  on  a  single  specific  property 
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Strengths  of  Dynamic  Analysis 


•  No  need  for  code 

•  Conceptually  easier  -  “if  you  can  run  the 
system,  you  can  run  the  test”. 

•  No  (less)  need  to  build  or  validate  modeis 
or  make  assumptions. 

•  Checks  installation  and  operation,  along 
with  end-to-end  or  whole-system. 
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Static  and  Dynamic  Analysis 
Complement  Each  Other 


Static  Analysis 


Dynamic  Analysis 


•  Handles  unfinished  code 

•  Can  find  backdoors,  eg, 
fuii  access  for  user  name 
“JoshuaCaieb” 

•  Potentiaiiy  compiete 


Code  not  needed,  eg, 
embedded  systems 

Has  few(er)  assumptions 

Covers  end-to-end  or 
system  tests 

Assess  as-instaiied 
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•  Assurance  that  software  is  less 
vulnerable  to  coming  cyberassaults 

•  Static  and  dynamic  analysis 

•  Static  Analysis  Tool  Exposition  - 
2009  outcomes  and  2010  progress 
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Static  Analysis  Tool  Exposition 
(SATE)  Overview 

•  Goal:  advance  research  in,  and  improvement  of, 
static  analysis  tools  for  security-relevant  defects 
and  speed  tool  adoption  by  demonstrating  use  on 
real  software. 

•  Checkpoints 

-  Participants  run  tools  on  Java  and  C  programs  we  choose 

-  NIST-led  researchers  analyze  reports 

-  Everyone  shares  results  and  observations  at  a  workshop 

-  Later  release  final  report  and  all  data 

•  http://samate.nist.gov/SATE.html 

•  Co-funded  by  NIST  and  DHS/NCSD 
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SATE  Participants 


•  2008: 

•  Aspect  Security  ASC 

•  Checkmarx  CxSuite 

•  Flawfinder 

•  Fortify  SCA 

•  Grammatech  CodeSonar 

•  2009: 

•  Armorize  CodeSecure 

•  Checkmarx  CxSuite 

•  Coverity  Prevent 

•  Grammatech  CodeSonar 


HP  Devinspect 
SofCheck  inspector  for  Java 
UMD  FindBugs 
Veracode  Security  Review 


Kiocwork  insight 
LDRA  Testbed 
SofCheck  inspector  for  Java 
Veracode  Security  Review 
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“Number  of  bugs”  is  undefined 

Tangled  Flow:  2  sources,  2  sinks,  4  paths 
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Summary  of  2009  tool  reports 


•  Reports  from  18  tool  runs 

•  About  20,000  total  warnings 

-  but  tools  prioritize  by  severity,  likelihood 

•  Reviewed  521  warnings  -  370  were  not  false 

•  Number  of  warnings  varies  a  lot  by  tool  and 
case 

•  83  CWE  ids/221  weakness  names 

26  April  2010  Paul  E.  Black  23 

I  I  National  Institute  of  Standards  and  Technology  •  U.S.  Department  of  Commerce 


Tools  don’t  report  same  warnings 


26  April  2010 


Overlap  in  Not-False  Warnings 


□  1  tool 

□  2  tools 

□  3  tools 
■  4  tools 


NIST 


Paul  E.  Black 


24 


National  Institute  of  Standards  and  Technology  •  U.S.  Department  of  Commerce 


Some  types  have  more  overlap 
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Overlap  in  Not-False  Buffer  Errors 


□  1  tool 

□  2  tools 

□  3  tools 
■  4  tools 
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Why  don’t  tools  find  same  things? 


•  Tools  look  for  different  weakness  classes 

•  Tools  are  optimized  differently 


- ^ 

more  severe 
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Tools  find  things  that  people  find 


hard  for  tools 
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SATE  2010  tentative  timeline 


Hold  organizing  workshop  (12  Mar  2010) 

Recruit  pianning  committee. 

•  Revise  protocoi. 

•  Choose  test  sets.  Provide  them  to  participants 
(17  May) 

•  Participants  run  their  toois.  Return  reports  (25 
June) 

•  Anaiyze  tool  reports  (27  Aug) 

•  Share  results  at  workshop  (October) 

•  Publish  data  (after  Jan  201 1 ) 
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Acronyms 


•  CWE  -  Common  Weakness  Enumeration 
http://cwe.mitre.com/ 

•  DHS/NCSD  -  Department  of  Homeland 
Security/National  Cyber  Security  Division 

•  MC/DC  -  Modified  Condition/Decision  Coverage 

•  SAMATE  -  Software  Assurance  Metrics  And  Tool 
Evaluation  (project  at  NIST) 

•  SATE  -  Static  Analysis  Tool  Exposition  (annual 
event) 

•  NIST  -  National  Institute  of  Standards  and 
Technology 
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